Frontplane communication network between multiple pluggable modules on a single faceplate

ABSTRACT

A pluggable module includes an interface configured to connect to a connector associated with a device having internal circuits; an optical interface configured to connect to a network path, to enable communication between the internal circuits and the network path; and a signal interface configured to connect to a second pluggable module in the device, wherein the signal interface connects to the second pluggable module as an out-of-band channel that operates independently from a backplane of the device and the internal circuits.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present disclosure is a continuation of U.S. patent application Ser.No. 14/925,929, filed Oct. 28, 2015, and entitled “Frontplanecommunication network between multiple pluggable modules on a singlefaceplate,” the contents of which are incorporated herein by reference.

BACKGROUND

A network (e.g., a telecommunication network) may have a data plane thatincludes devices exchanging network traffic, and a control plane thatcontrols how network traffic is routed in the data plane. Operationssupport systems (OSSs) in the control plane are computer systems used bytelecommunications service providers to manage telecommunicationnetworks, such as telephone networks. OSSs and data plane devices(referred to as Network Elements (NEs)) exchange messages to performmanagement functions, such as network inventory, service provisioning,network configuration, and fault management, etc. For example, themessages may be based on Transaction Language 1 (TL1), which is a widelyused management protocol in telecommunications. In particular,operations domains (e.g., surveillance, memory administration, andaccess and testing) define and use TL1 messages to accomplish specificfunctions between the OSSs and the NEs.

A message (e.g., TL1 message) may include a command, which is adirective to a computer program, in a receiving device of the message,to perform a specific task. The command that specifies a task to beperformed in a network may be referred to as a network command. Themessage that includes the network command may be referred to as anetwork command message. Typically, the receiving device executes (orotherwise processes) the command immediately after receiving the messageand also reports the associated response immediately. When the commandis sent to a large number of receiving devices, problems may arise dueto the timing difference (referred to as a command skew) with respect towhen the command is received by any two receiving devices. For example,when a command is sent to more than 10 NEs in a telecommunicationnetwork, the command skew produces unreliable results of the command.

IEEE 1588 standards describe a hierarchical master-slave architecturefor clock distribution. Under this architecture, a clock distributionsystem consists of one or more communication media (network segments)and one or more clocks. An ordinary clock is a device with a singlenetwork connection and is either the source (master) of or destination(slave) for a synchronization reference. A boundary clock is a devicewith multiple network connections and can accurately synchronize onenetwork segment to another. A synchronization master is a deviceselected for each of the network segments in the system. The root timingreference is called the grandmaster. The grandmaster is a device thattransmits synchronization information to the clocks residing on itsnetwork segment. The boundary clocks with a presence on that segmentthen relay accurate time to the other segments to which they are alsoconnected. IEEE 1588-2008 introduces a transparent clock associated withnetwork equipment used to convey PTP (precision timing protocol)messages. The transparent clock modifies PTP messages as they passthrough the device. In particular, timestamps in the PTP messages aremodified to correct for communication latency, which is time spenttraversing the network equipment. This scheme improves distributionaccuracy by compensating for delivery variability across the network.

SUMMARY

In general, in one aspect, a pluggable module includes an interfaceconfigured to connect to a connector associated with a device havinginternal circuits; an optical interface configured to connect to anetwork path, to enable communication between the internal circuits andthe network path; and a signal interface configured to connect to asecond pluggable module in the device, wherein the signal interfaceconnects to the second pluggable module as an out-of-band channel thatoperates independently from a backplane of the device and the internalcircuits.

In general, in another aspect, a device includes a backplane; internalcircuits communicatively connected to the backplane; one or moreconnectors communicatively coupled to the internal circuit and thebackplane, wherein the one or more connectors are each configured toconnect to a pluggable module of a plurality of pluggable modules,wherein the plurality of pluggable modules are configured to communicateto one another via an in-band channel through the device and via anout-of-band channel that operates independently from the device.

In general, in a further aspect, a method includes operating a devicewith a plurality of pluggable modules; communicating between two or morepluggable modules of the plurality of pluggable modules via an in-bandchannel through the device; and communicating between the two or morepluggable modules via an out-of-band channel that operates independentfrom and bypasses the device.

In general, in one aspect, the invention relates to a system. The systemincludes (i) a device including connectors connected to pluggablemodules external to the device and (ii) the pluggable modules exchangingsignals with the device via the connectors, where the pluggable modulesinclude a first pluggable module and a second pluggable module thatfurther exchange a supplemental signal with each other and bypassing theconnectors.

In general, in one aspect, the invention relates to a system. The systemincludes (i) a device including a backplane internal to the device, andconnectors connected to pluggable modules external to the device, wherethe connectors hold the pluggable modules in a rigid relative positionto the device with mechanical stability, and (ii) the pluggable modulesexchanging signals with the device via the connectors and the backplaneaccording to a pre-determined communication protocol. The system furtherincludes (iii) a communication assembly peripheral to the device and inproximity of the connectors, where the communication assembly isconfigured to exchange a supplemental signal among the pluggable modulesand bypassing the connectors and the backplane, and compensate acommunication latency among the pluggable modules and the device byusing the supplemental signal to enhance the pre-determinedcommunication protocol.

In general, in one aspect, the invention relates to a method. The methodincludes connecting, using connectors of a device, the device topluggable modules external to the device, where the device includes abackplane internal to the device. The method further includes holding,using the connectors, the pluggable modules in a rigid relative positionto the device with mechanical stability, exchanging, based on apre-determined communication protocol, signals among the pluggablemodules and the device via the connectors and the backplane, furtherexchanging, via a communication assembly peripheral to the device and inproximity of the connectors, a supplemental signal among the pluralityof pluggable modules and bypassing the connectors and the backplane, andreducing a communication latency among the pluggable modules by usingthe supplemental signal to enhance the pre-determined communicationprotocol.

Other aspects of the invention will be apparent from the followingdescription and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of a system in accordance with one or moreembodiments of the invention.

FIG. 2 depicts prior art pluggable modules plugging into a device.

FIG. 3 shows a schematic diagram of communication channels of pluggablemodules plugged into a device in accordance with one or more embodimentsof the invention.

FIG. 4 depicts pluggable modules and a communication assembly that areassociated with a device in accordance with one or more embodiments ofthe invention.

FIG. 5 depicts pluggable modules that are plugged into a device andcoupled to a communication assembly in accordance with one or moreembodiments of the invention.

FIG. 6 shows schematic diagrams of examples of the communicationassembly in accordance with one or more embodiments of the invention.

FIG. 7 shows a flowchart of a method in accordance with one or moreembodiments of the invention.

FIG. 8 shows a computer system in accordance with one or moreembodiments of the invention.

DETAILED DESCRIPTION

Specific embodiments of the invention will now be described in detailwith reference to the accompanying figures. Like elements in the variousfigures are denoted by like reference numerals for consistency.

In the following detailed description of embodiments of the invention,numerous specific details are set forth in order to provide a morethorough understanding of the invention. However, it will be apparent toone of ordinary skill in the art that the invention may be practicedwithout these specific details. In other instances, well-known featureshave not been described in detail to avoid unnecessarily complicatingthe description.

In general, embodiments of the invention provide a system and a methodfor providing an out-of-band channel for pluggable modules external to adevice. In particular, the pluggable modules exchange signals with thedevice via connectors of the device. In addition, the pluggable modulesfurther exchange a supplemental signal via the out-of-band channel andbypassing the connectors. In one or more embodiments of the invention,the device connects to network paths in a computer network where thepluggable modules provide an interface function between the device andthe network paths. Accordingly, network messages traverse the networkpaths through the device according to a pre-determined communicationprotocol. A communication latency occurs when the network messagestraverse through the device based on the signals exchanged between thedevice and the pluggable modules.

In one or more embodiments of the invention, the network messagestraverse through the device via the connectors and a backplane internalto the device. In one or more embodiments of the invention, theconnectors hold the pluggable modules in a rigid relative position tothe device with mechanical stability. In addition, a communicationassembly is peripheral to the device and in proximity of the connectors.Specifically, the communication assembly is configured to exchange thesupplemental signal among the pluggable modules and bypassing theconnectors and the backplane. Accordingly, the communication assemblycompensates the aforementioned communication latency by using thesupplemental signal to enhance the pre-determined communicationprotocol.

FIG. 1 shows a system (100) in accordance with one or more embodiments.As shown in FIG. 1, the system (100) includes multiple devices (e.g.,device X (106X), device Y (106Y), device Z (106Z), etc.) connected by acomputer network (150) in accordance with one or more embodiments. Inparticular, the multiple devices (e.g., device X (106X), device Y(106Y), device Z (106Z), etc.) are connected via network paths (e.g.,network path A (151), network path B (152), etc.) of the computernetwork (150). In one or more embodiments, a device (e.g., device X(106X)) is connected to a network path (e.g., network path A (151),network path B (152)) via a pluggable module (e.g., pluggable module AX(105AX), pluggable module BY (105BY), etc.) that provides an interfacefunction between the device and the network path. In addition, acommunication assembly (e.g., communication assembly (107)) may bedisposed in a peripheral position of or optionally attached to a singledevice (e.g., device X (106X)). In one or more embodiments, thecommunication assembly (107) allows communications bypassing the deviceX (106X) and among pluggable modules (e.g., pluggable module AX (105AX),pluggable module BY (105BY), etc.) connected to the device X (106X).

In one or more embodiments of the invention, the computer network (150)includes wired and/or wireless portions of a telecommunications network.In one or more embodiments, a telecommunications network includes acollection of devices, links, and any intermediate nodes connected toenable telecommunication between the devices of the telecommunicationsnetwork. The device X (106X), device Y (106Y), and device Z (106Z) arenodes of the telecommunication network and may be computing devices ornetworking devices of any type, including servers, routers, switches,mobile phones, desktop or tablet personal computers (PCs), etc. Withinthe computer network (150), a network path (e.g., network path A (151),network path B (152)) between two devices is a sequence of links,intermediate nodes, and/or any intervening devices that connect the twodevices. In one or more embodiments, a device may be connected tomultiple network paths. For example, the device X (106X) is connected tothe network path A (151) and the network path B (152) via the pluggablemodule AX (105AX) and the pluggable module BY (105BY), respectively. Inone or more embodiments, multiple network paths may exist between twodevices. Accordingly, messages or other telecommunication data areexchanged between the two devices via one or more of the network paths.The transmission delay of sending/receiving messages over any particularnetwork path (e.g., network path A (151), network path B (152)) maydepend on characteristics of the links, intermediate nodes, and/or anyintervening devices contained in the particular network path.

In one or more embodiments of the invention, the device X (106X), deviceY (106Y), and device Z (106Z) are configured to send and/or receivenetwork messages over the computer network (150). The device X (106X),device Y (106Y), and device Z (106Z) may include controlling devices inthe control plane of the telecommunication network and/or NEs in thedata plane of the telecommunication network. In one or more embodiments,the network messages may be based on TL1 or other management protocols.In one or more embodiments, the network messages transmit across thenetwork path A (151) and network path B (152) according to acommunication protocol or a network protocol (e.g., Ethernet,Synchronous Optical Networking (SONET)/Synchronous Digital Hierarchy(SDH), G.709 Optical Transport Network (OTN), etc.) of the computernetwork (150).

In one or more embodiments of the invention, the device X (106X)includes a backplane (101), internal circuits (102), and connectors(e.g., connector X (105X), connector Y (105Y)). In one or moreembodiments, the backplane (101), internal circuits (102), andconnectors (e.g., connector X (105X), connector Y (105Y)) aresubstantially enclosed within a mechanical enclosure (not shown) of thedevice X (106X). These components of the device X (106X) are describedbelow. In one or more embodiments, the backplane (101) may be omitted.

A connector is an electro-mechanical device for joining electricalcircuits using a mechanical assembly. As used herein, the term“electro-mechanical” refers to a combination of “electrical” and‘mechanical.” Further, the term “electrical” may refer tooptoelectronics and/or electro-optics properties. In one or moreembodiments, an electrical circuit may optionally include anoptoelectronic circuit and/or electro-optics circuit. Two electricalcircuits may be connected via an electrical connection and a mechanicalconnection provided by the electro-mechanical device. In one or moreembodiments, the connector X (105X) and connector Y (105Y) providetemporary connections between the internal circuits (102) and thepluggable modules (e.g., pluggable module AX (105AX), pluggable moduleBY (105BY)) external to the mechanical enclosure of the device X(106X)). In particular, each external pluggable module is connected tothe device X (106X), more specifically the internal circuits (102), whenthe electrical connection and the mechanical connection are made using acorresponding connector (e.g., connector X (105X), connector Y (105Y)).From time to time, each external pluggable module may be disconnectedfrom the device X (106X) or the internal circuits (102) by separating(e.g., pulling apart) from the corresponding connector (e.g., connectorX (105X), connector Y (105Y)). Specifically, separating from thecorresponding connector disengages the electrical connection and themechanical connection. In one or more embodiments, the mechanicalassembly of the connector X (105X) is adapted to hold, when connected,the pluggable module AX (105AX) in a rigid relative position to thedevice X (106X) with mechanical stability. As used herein, the term“mechanical stability” refers to a pre-determined mechanical toleranceof location, orientation, alignment, or other mechanical property of thepluggable module AX (105AX) relative to the device X (106X). Further,the term “rigid” refers to a limited range of movement of the pluggablemodule AX (105AX) relative to the device X (106X) over time. In one ormore embodiments, the pre-determine mechanical tolerance and the limitedrange of movement maintains an acceptable distance and alignment betweentwo or more signal interfaces (described below) disposed on thepluggable module AX (105AX) and the device X (106X) for proximity basedsignal exchange. For example, the rigid relative position may bemaintained by an interlocking mechanism between each pluggable moduleand the corresponding connector. The interlocking mechanism may includea locking pin or tab that interlocks a male portion and a female portionof the mechanical connection. Similarly, the mechanical assembly of theconnector Y (105Y) is adapted to hold, when connected, the pluggablemodule BY (105BY) in another rigid relative position to the device X(106X) with mechanical stability. Accordingly, the pluggable module AX(105AX) and the pluggable module BY (105BY), when connected to theconnectors of the device X (106X), are held in a rigid relative positionto each other. In one or more embodiments, the rigid relative positionbetween all pluggable modules (e.g., pluggable module AX (105AX),pluggable module BY (105BY), etc.) connected to the device X (106X)causes all such connected pluggable modules to be within apre-determined 3-dimensional (3D) range (e.g., 5 centimeters by 10centimeters by 60 centimeters, 60 centimeter on each of X-axis, Y-axis,and Z-axis, etc.). In particular, the pre-determined 3D range isdependent on a layout of the connectors on the mechanical enclosure ofthe device X (106X). In contrast, one or more of the pluggable modules,when being disconnected from the connectors of the device X (106X), mayexit from the pre-determined 3D range.

In one or more embodiments, each pluggable module (e.g., pluggablemodule AX (105AX), pluggable module BY (105BY), etc.) is anelectro-mechanical device for providing an interface function betweenthe internal circuits (102) and the network paths (e.g., network path A(151), network path B (152)). In one or more embodiments, the interfacefunctions of the pluggable module AX (105AX) and the pluggable module BY(105BY) allow network communication data packets (e.g., forming anetwork message) to be routed between the network path A (151) and thenetwork path B (152) through the device X (106X). Specifically, thepluggable module AX (105AX) and the pluggable module BY (105BY) exchangeone or more signals with the device X (106X) via the connector X (105X)and connector Y (105Y) to route the network communication data packets.For example, the one or more signals may traverse the connector X(105X), the internal circuits (102), the backplane (101), and theconnector Y (105Y) to route the network communication data packets. Anexample of routing the network communication data packets via the deviceX (106X) is described in reference to FIG. 3 below.

Returning to the discussion of the device X (106X), in one or moreembodiments, the internal circuits (102) include hardware circuitriesand embedded software components for implementing one or morefunctionalities of the device X (106X). The backplane (101) is used as abackbone to connect several portions (e.g., printed circuit boards, flexcircuit, etc.) of the internal circuits (102) together to make up acomplete system. In one or more embodiments, the backplane (101)includes a group of electrical connectors in parallel with each other,so that each pin of each connector is linked to the same relative pin ofall the other connectors forming a computer bus. The electricalconnectors on the backplane (101) are internal connectors that areseparate and distinct from the aforementioned connectors for connectingthe pluggable modules. In one or more embodiments, the internal circuits(102) are self-contained without any backplane for connection. In otherwords, the backplane (101) may be an integral part of the internalcircuits (102) or may be omitted entirely. Additional details of thedevice X (106X) are described in reference to FIG. 2 below.

In one or more embodiments, the communication assembly (107) isperipheral to the device X (106X) and in proximity of the connectors ofthe device X (106X). In particular, the communication assembly (107) isexternal to the mechanical enclosure of the device X (106X). Forexample, the communication assembly (107) may be disposed in anintervening position between the pluggable modules connected to thedevice X (106X) and the mechanical enclosure of the device X (106X). Inone or more embodiments, the communications assembly (107) may not existas a physical entity. For example, the communications assembly (107) maycorrespond to a free space allowing wireless communication therethrough.An example of the communication assembly (107) disposed in suchintervening position is described in reference to FIGS. 4 and 5 below.

As note above, the communication assembly (107) allows communicationsamong the pluggable modules (e.g., pluggable module AX (105AX),pluggable module BY (105BY), etc.) connected to the device X (106X). Inone or more embodiments, the communication assembly (107) is configuredto exchange a supplemental signal (not shown) among the pluggablemodules (e.g., pluggable module AX (105AX), pluggable module BY (105BY),etc.) and bypassing the device X (106X). In particular, the supplementalsignal is exchanged among the pluggable modules (e.g., pluggable moduleAX (105AX), pluggable module BY (105BY), etc.) without traversing theconnectors (e.g., connector X (105X), connector Y (105Y)), the internalcircuits (102), and the backplane (101) of the device X (106X). In oneor more embodiments, the supplemental signal exchanged among thepluggable modules (e.g., pluggable module AX (105AX), pluggable moduleBY (105BY), etc.) is used to compensate a communication latency amongthe pluggable modules (e.g., pluggable module AX (105AX), pluggablemodule BY (105BY), etc.). In particular, the communication latency iscompensated by using the supplemental signal to enhance a communicationprotocol used to route the network communication data packets via thedevice X (106X). Throughout this disclosure, the term “supplementalsignal” refers to the signal exchanged among the pluggable moduleswithout traversing the connectors or other portion of the device thatthe pluggable modules are connected to.

Although FIG. 1 only shows three devices (i.e., device X (106X), deviceY (106Y), device Z (106Z)) and two network paths (i.e., network path A(151), network path B (152)), those skilled in the art, having thebenefit of this detailed description, will appreciate that the system(100) may have any number of devices and network paths. Further,although FIG. 1 only shows one communication assembly (107) and twopluggable modules (i.e., pluggable module AX (105AX), pluggable moduleBY (105BY)), those skilled in the art, having the benefit of thisdetailed description, will appreciate that the system (100) may have anynumber of communication assemblies and pluggable modules. In one or moreembodiments of the invention, the operations of the system (100) areperformed using the method described in reference to FIG. 7 below.

FIG. 2 depicts a prior art pluggable modules plugging into a device. Asshown in FIG. 2, the device (200) having multiple connectors (e.g.,connector A (231), connector B (232), etc.) and the backplane (230) isan example of the device X (106X) having multiple connectors (e.g.,connector X (105X), the connector Y (105Y), etc.) and the backplane(101) depicted in FIG. 1 above. In addition, the optical fiber A (211)and the optical fiber B (212) are example elements included in thenetwork path A (151) and the network path B (152) depicted in FIG. 1above. Further, the pluggable module A (201) and the pluggable module B(202) are examples of the pluggable module AX (105AX) and the pluggablemodule BY (105BY) that provide an interface function between the device(200) and the optical fibers (i.e., optical fiber A (211), optical fiberB (212)). In particular, the arrows (221) and (222) represent connectingthe pluggable module A (201) and the pluggable module B (202) to theconnector A (231) and the connector B (232), respectively. Although notexplicitly shown, the optical fiber A (211) and the optical fiber B(212) may be connected to the pluggable module A (201) and the pluggablemodule B (202). As noted above, the connections from the pluggablemodule A (201) and the pluggable module B (202) to the connector A (231)and the connector B (232) may include electrical-only, optical-only,optoelectrical, and/or electro-optical connections. Similarly, theconnections from the optical fiber A (211) and the optical fiber B (212)to the pluggable module A (201) and the pluggable module B (202) mayalso include electrical-only, optical-only, optoelectrical, and/orelectro-optical connections.

As noted above, the network messages may traverse the network paths(e.g., optical fiber A (211), optical fiber B (212)) through the device(200) according to a pre-determined communication protocol. Thepre-determined communication protocol may include or otherwise based onthe IEEE 1588 standards, the precision timing protocol, Ethernet,Synchronous Optical Networking (SONET)/Synchronous Digital Hierarchy(SDH), G.709 Optical Transport Network (OTN), etc. In particular, theIEEE 1588 standards provide timing synchronization for devices in anetwork by aligning “slave clocks” for sync-capable devices. This isachieved by first deploying a grandmaster clock that all slave clocks inthe network align to. In the IEEE 1588 context, the terms “grandmasterclock” and “slave clocks” may refer to the actual clock signals or thedevices using the clock signals interchangeably. In this context, thegrandmaster clock exchanges a series of messages (e.g., Sync, Follow-up,Delay Request and Delay Response) between each slave clock. Thesemessages allow the slave clocks to calculate the delay in a network andcompensate for the delay when the slave clock is being aligned. In otherwords, when the grandmaster clock announces the exact time at any givenmoment to a slave clock, the slave clock computes the time period forthe announcement to reach itself in order to accurately align its clockwith the grandmaster clock.

In one scenario of implementing the IEEE 1588 standards, the device(200) includes special features such as accurate time stampcapabilities, special software to recover and analyze IEEE 1588 packets,an oven-controlled oscillator and specialized hardware that implements aphase locked loop. Specifically, the device (200) uses these specialfeatures to correct, regenerate, or otherwise modify an IEEE 1588 datastream and forward a newly constructed data stream that corrects for theasymmetric delay on the backplane (230) of the device (200).

In another scenario of implementing the IEEE 1588 standards, the device(200) provides frequency and phase references to the pluggable module A(201) and the pluggable module B (202). These references are used todrive timestamp circuits inside the pluggable module A (201) and thepluggable module B (202) to remove or otherwise compensate thecommunication latency due to noisy residence time within the device(200).

In either scenario of implementing the IEEE 1588 standards describedabove, communication between two pluggable modules (e.g., the pluggablemodule A (201) and pluggable module B (202)) is routed through thedevice (200) with an asymmetric timing delay particular to thecorresponding connector pair. Packet delay variation between theasymmetric timing delays of different connector pairs negatively impactsaccuracy of the timestamper circuits in the corresponding pluggablemodules. Accordingly, delay-critical communication between two pluggablemodules (e.g., pluggable module A (201), pluggable module B (202)) isnot possible. As a result, when a network is scaled up and/or softwaredefined networks are deployed, the packet delay variation from theasymmetric timing delays may render a clock distribution systemunusable.

FIG. 3 shows a schematic diagram of communication channels of pluggablemodules plugged into a device in accordance with one or more embodimentsof the invention.

In telecommunications and computer networking, a communication channelor channel refers to either a physical transmission medium such as awire, or to a logical connection over a multiplexed medium such as aradio channel. A channel is used to convey an information signal, forexample, a digital bit stream, from one or more senders (ortransmitters) to one or more receivers. A channel has a certain capacityfor transmitting information, often measured by its bandwidth in Hz orits data rate in bits per second.

As shown in FIG. 3, the device (300) having multiple connected pluggablemodules (e.g., pluggable module X (301), pluggable module Y (302), etc.)and the backplane (320) is an example of the device X (106X) havingmultiple connected pluggable modules (e.g., pluggable module AX (105AX),pluggable module BY (105BY), etc.) and the backplane (101) depicted inFIG. 1 above. In addition, the Ethernet switch (310) is an example ofthe internal circuits (102) of the device X (106X) depicted in FIG. 1above.

Further as shown in FIG. 3, the in-band channel (350) represents thephysical transmission medium that allow network messages to travelthrough the device (300) according to the aforementioned communicationprotocol(s). The timing delay incurred by a network message or by asignal to travel through the device (300) from the pluggable device X(301) to the pluggable device Y (302) via the Ethernet switch (310), thebackplane (320) and any intervening signal path is referred to as acommunication latency of the two pluggable modules. The communicationlatency for different pairs of pluggable modules of the device (300) mayvary. The variation in the communication latency for different pairs ofpluggable modules is referred to as a time delay variation of the device(300). In one or more embodiments of the invention, the device (300)and/or the pluggable module X (301) and pluggable module Y (302) may notinclude all the aforementioned special features for implementing theIEEE 1588 standards. In such embodiments, the pluggable module X (301)and pluggable module Y (302) include the signal interface X (311) andsignal interface Y (312), respectively, to implement an out-of-bandchannel (360). In one or more embodiments, the out-of-band channel (360)is used to exchange a supplemental signal among the connected pluggablemodules of the device (300). Specifically, the out-of-band channel (360)and the supplemental signal supplements the aforementioned communicationprotocol(s) to meet the timing requirement of the clock distributionsystem. For example, the supplemental signal may provide asynchronization timing that allows the communication latency and/or timedelay variation to be compensated.

In an example scenario, an IEEE 1588 grandmaster sends a timing packetstream to the device (300), which is an old legacy node not fullycompliant with the IEEE 1588 or SyncE (Synchronous Ethernet) standards.The pluggable module X (301) and pluggable module Y (302) exchange thesupplemental signal via the out-of-band channel (360) to establish acommon frequency and phase, referred to as a common time base or asingle time base. This single time base may be a low bandwidth (e.g., 64baud) signal and is independent of or unrelated to the grandmaster'sfrequency and phase. The timing packet stream from the grandmaster isthen processed as below.

The pluggable module X (301) receives the grandmasters timing packetstream and performs ingress timestamping before passing the timingpacket stream to the pluggable module Y (302) through the in-bandchannel (350). Accordingly, the pluggable module Y (302) receives thetiming packet stream and performs egress timestamping. The ingresstimestamp represents or otherwise corresponds to a time point when thetiming packet stream is received by the pluggable module X (301). Theegress timestamp represents or otherwise corresponds to a time pointwhen the timing packet stream is received by the pluggable module Y(302). The ingress and egress timestamps are both referenced to thesingle time base enabled by the out-of-band channel (360). The ingressand egress timestamps are subtracted or otherwise compared to generate adifference (referred to as the residence time) that is used to updatethe correction field in the timing packet stream. The timestamps may besubtracted or otherwise compared because they are both referenced to thesingle time base enabled by the out-of-band channel (360). Without theout-of-band channel (360), the pluggable module X (301) and pluggablemodule Y (302) would establish another common time base through thein-band channel (350) where a noise level of the Ethernet switch (310)may degrade timing accuracy. By using the out-of-band channel (360),network timing may be moved off the in-band channel (350) to eliminateasymmetric delays and packet delay variations and increase performanceand scalability.

In one or more embodiments, the out-of-band channel (360) is used as amanagement channel for diagnostic purposes to bypass the network device(350) that may become inoperable. For example, in the computer network(150) depicted in FIG. 1, the device Y (106Y) and device Z (106Z) mayuse the out-of-band channel (if available) via the pluggable module AX(105AX) and pluggable module BY (105BY) to communicate for diagnosticpurposes in the event that the device X (106X) may become inoperable. Inanother example, the pluggable module AX (105AX) and/or correspondingport of the device X (106X) may be disabled via the out-of-band channel(if available) in the event that the device X (106X) may becomeinaccessible due to a denial-of-service (DoS) attack.

In one or more embodiments, the in-band channel (350) supportscommunication between devices that are kilometers apart in the computernetwork (150). Although a single out-of-band channel supportscommunication between pluggable modules within a much shorter range(e.g., 60 centimeters), multiple out-of-band channels may be coupled viathe network paths to form a larger out-of-band channel that spans alarge part of the network or the entire network.

FIG. 4 depicts pluggable modules and a communication assembly that areassociated with a device in accordance with one or more embodiments ofthe invention. Specifically, FIG. 4 shows a 3D exterior view of thedevice (300) depicted in FIG. 3 above.

In one or more embodiments of the invention, the device (300) may besimilar to the device (200) depicted in FIG. 2 above and uses similartypes of connectors (e.g., connector A (231), connector B (232), etc.)and similar types of optical fibers (e.g., optical fiber A (211),optical fiber B (212), etc.). In addition, the backplane (320) of thedevice (300) may be similar to the backplane (230) of the device (200)depicted in FIG. 2 above. In one or more embodiments, the device (300)may not include all the aforementioned special features for implementingthe IEEE 1588 standards as the device (200). Instead, the pluggablemodule X (301) and pluggable module Y (302) include the signal interfaceX (311) and signal interface Y (312) to implement the out-of-bandchannel (360) depicted in FIG. 3 above. In one or more embodiments,other than having the additional signal interface X (311) and signalinterface Y (312), the pluggable module X (301) and pluggable module Y(302) are similar to the pluggable module A (201) and pluggable module B(202) depicted in FIG. 2 above. In one or more embodiments, in contrastto the pluggable module A (201) and pluggable module B (202) depicted inFIG. 2 above, the timestamper circuits are omitted from the pluggablemodule X (301) and pluggable module Y (302).

In one or more embodiments, the signal interface X (311) and signalinterface Y (312) include free-space short-range optical communicationcircuitries, with and without the help of a wave-guide, to implement theout-of-band channel (360) depicted in FIG. 3 above. In particular, thefree-space short-range optical communication circuitries are configuredto establish data communication within the pre-determined 3D range(described in reference to FIG. 1 above) that is defined by a layout ofthe connectors (e.g., connector A (231), connector B (232), etc.) on themechanical enclosure of the device (300). Accordingly, theaforementioned supplemental signal may be exchanged among the pluggablemodules using the free-space short-range optical communicationcircuitries.

In one or more embodiments, the signal interface X (311) and signalinterface Y (312) include free-space short-range wireless communication(e.g., based on NFC, Bluetooth, Wi-fi, etc.) circuitries to implementthe out-of-band channel (360) depicted in FIG. 3 above. In particular,the free-space short-range wireless communication circuitries areconfigured to establish data communication within the pre-determined 3Drange (described in reference to FIG. 1 above) that is defined by alayout of the connectors (e.g., connector A (231), connector B (232),etc.) on the mechanical enclosure of the device (300). Accordingly, theaforementioned supplemental signal may be exchanged among the pluggablemodules using the free-space short-range wireless communicationcircuitries.

In one or more embodiments, the signal interface X (311) and signalinterface Y (312) are coupled using a stranded single-conductor cable toimplement the out-of-band channel (360) depicted in FIG. 3 above. Theexisting ground path of the pluggable module X (301) and pluggablemodule Y (302) is used as signal return. An encoding scheme thatspecifically supports the desired application (e.g., delay accuracy fortiming applications) is used to transmit/receive data over theout-of-band channel. The stranded single-conductor cable may beclear-taped to an existing faceplate of the device (300) withoutobscuring printed details on the faceplate. Specifically, the faceplateis one side of the mechanical enclosure where the connectors arelocated.

In one or more embodiments, the out-of-band channel (360) depicted inFIG. 3 above is implemented using the communication assembly (450). Inone or more embodiments, the communication assembly (450) is adhered tothe device (300) and being adjacent to or overlapping the connector A(231) and connector B (232). For example, the communication assembly(450) may be adhered to the device (300) using velcro, glue, stickytape, magnet, press-fit, snap-on, or other permanent or semi-permanentmechanism. In one or more embodiments, the communication assembly (450)may be positioned adjacent to the device (300) within the pre-determined3D range (described in reference to FIG. 1 above) that is defined by alayout of the connectors (e.g., connector A (231), connector B (232),etc.) on the mechanical enclosure of the device (300). In suchembodiments, the communication assembly (450) may not be adhered to thedevice (300) using any permanent or semi-permanent mechanism. As shownin FIG. 4, the connectors (e.g., connector A (231), connector B (232),etc.) are disposed in a frontal face (referred to as a faceplate) of thedevice (300). In this context and in contrast to the backplane (320),the communication assembly (450) may be referred to as a frontplane andthe out-of-band channel (360) may be referred to as a frontplanecommunication network.

With the communication assembly (450) adhered to or adjacent to thedevice (300), in one or more embodiments, a signal interface A (351) ofthe communication assembly (450) is disposed in proximity to theconnector A (231) and a signal interface B (352) of the communicationassembly (450) is disposed in proximity to the connector B (232). In oneor more embodiments, the pluggable module X (301) and pluggable module Y(302) may be connected to the connector A (231) and connector B (232) bypenetrating through openings (i.e., hole A (400), hole B (401)) on thecommunication assembly (450). The resultant rigid relative positions ofthe pluggable modules to the device (300) maintain the signal interfaceA (351) and signal interface B (352) to be in proximity to or inphysical connect with the signal interface X (311) and signal interfaceY (312), respectively. Additional details of maintaining such proximityor physical contact are described in reference to FIG. 5 below.Accordingly, the aforementioned supplemental signal may be exchangedbetween the signal interfaces on the communication assembly (450) withthe signal interfaces on the pluggable modules via proximity-basedconnections (e.g., optical-coupling, inductive-coupling,capacitive-coupling, or other wireless connections) or via physicalcontacts.

In one or more embodiments, multiple copies of the supplemental signalmay be exchanged using multiple connections based on the proximity-basedconnections or the physical contacts. These multiple copies of thesupplemental signal are redundant to each other in a failover operation.In other words, one copy of the supplemental signal may be automaticallyswitched over to a redundant copy of the supplemental signal upondetecting that the copy of the supplemental signal in use hasencountered a failure condition. For example, if the copy of thesupplemental signal currently in use is temporarily jammed, theredundant copy of the supplemental signal may be used. In one or moreembodiments, the multiple copies of the supplemental signal useorthogonal schemes and/or transmission media that are not susceptible tothe same interference.

In one or more embodiments, the communication assembly (450) includes arouting element configured to route the supplemental signal between thesignal interface A (351) and the signal interface B (352). Accordingly,the aforementioned supplemental signal may be exchanged among thepluggable modules via the signal interfaces and routing element on thecommunication assembly (450) and the signal interfaces on the pluggablemodules. In one or more embodiments, the communication assembly (450)includes a printed circuit board (PCB) and an optional cover forcosmetic or protective purpose. In such embodiments, the routing elementmay include copper traces on the PCB. In one or more embodiments, thecommunication assembly (450) is substantially transparent with theappropriate choice of dielectric materials (e.g., polycarbonate). Insuch embodiments, the routing element may include light pipes or lightguides made of the dielectric materials for transmitting a guidedoptical signal. Examples of the routing element are described inreference to FIG. 5 below.

In one or more embodiments, at least one of the signal interface A (351)and signal interface B (352) includes an active circuit that receiveswireless power transmission from the corresponding pluggable module. Forexample, the active circuit may receive power from a light emittingdiode (LED), a radio power transmission device, or an inductive powertransmission device of the corresponding pluggable module. In otherwords, the power transmission may use LED/photo cell pairs or a tunedresonant circuit with an inductive coupling mechanism. In one or moreembodiments, the aforementioned supplemental signal may provide bothpower transmission and data transmission.

FIG. 5 depicts pluggable modules that are plugged into a device andcoupled to a communication assembly in accordance with one or moreembodiments of the invention. Specifically, FIG. 5 shows the sameelements depicted in FIG. 4 except that the communication assembly (450)is explicitly shown as adhered to the device (300). In addition, thepluggable module X (301) and pluggable module Y (302) are plugged intothe connector A (231) and connector B (232) by penetrating the hole A(400) and hole B (401), respectively. The routing element described inreference to FIG. 4 is shown as the routing element (503) FIG. 5. Thecoupled signal interface X (501) represents the signal interface X (311)and signal interface A (351) in proximity or in physical contact to eachother. The coupled signal interface Y (502) represents the signalinterface Y (312) and signal interface B (352) in proximity or inphysical contact to each other. The signal interface X (311), signalinterface A (351), signal interface Y (312), and signal interface B(352) themselves as well as the connector A (231), connector B (232),optical fiber A (211), and optical fiber B (212) are not explicitlyshown in FIG. 5 for clarity.

In one or more embodiments, the communication assembly (450) is based onone or more of the following:

(a) A special single-conductor copper wire that connects two pluggablemodules and uses the ground of the device (300) as a return path. Thesingle-conductor copper wire may be attached to a faceplate of thedevice (300).

(b) A special optical wire that connects the pluggable modules andattached to the faceplate of the device (300).

(c) Wireless (e.g., NFC, Bluetooth, Wifi) communication.

(d) A bus connection.

(e) Capacitive coupling using only the ground connection of eachpluggable module to the device (300). The capacitive coupling mayoperate in a similar manner as gesture detection devices.

In one or more embodiments, the communication assembly (450) includessensors to communicate with appropriately-equipped pluggable modules.These sensors may be photo diodes for optical communication or coils fornear field communication (NFC). This communication may be one way or twoway. For example, one way communication may be used for timing transferwhile two way communication may allow reporting of the number andquantity of synchronized pluggable modules, the synchronization state,etc.

In one or more embodiments, the communication assembly (450) includes areceiver (or a transceiver for two-way communication). The receiver maybe used to detect the aforementioned supplemental signal distributedfrom a selected pluggable module acting as a central distributiontransmitter. Multiple communication assemblies associated with multipledevices may be in proximity to each other, such as mounted on anequipment rack. In such configuration, multiple central distributiontransmitters of these communication assemblies may employ a specificorthogonal spreading code, such as a Gold code, in a spread spectrumsignal to minimize interference. The spread spectrum signal may be maderobust to hacking attacks by employing a stream cipher to encrypt thetiming information, which may be recovered by correlation techniques.

In one or more embodiments, the communication assembly (450) iscompletely passive and provides a network interconnect for the pluggablemodules to communicate. The coupled signal interface may be based oncoils constructed within PCB layers of the communication assembly (450).No security mechanisms may be employed as communication occurs only inthe near field. Timing distribution/synchronization among the pluggablemodules may be performed using the supplemental signal with the properfrequency (e.g., 1 Hz) and phase. The supplemental signal may have asine wave format or square wave with controlled edge rates to mitigateelectromagnetic interference.

In one or more embodiments, the communication assembly (450) is entirelymade from a clear polycarbonate material that acts as a light guide orlight pipe for transmitting a guided optical signal. The light guide mayhave a symmetrical N:N splitter functionality that allows any-to-anyoptical communication between pluggable modules. The light guide mayalso interface with existing LED's on the pluggable modules and/or thefaceplate of the device (300) to collect system/port status indicated bythese LED's. In turn, the light guide allows the collected system/portstatus to be shared by the device (300) and the pluggable modules.

FIG. 6 shows schematic diagrams of examples of the communicationassembly in accordance with one or more embodiments of the invention. Asshown in FIG. 6, the communication assembly A (601) includes a busconnection based routing element A (611). The communication assembly B(602) includes a switched connection based routing element B (612) wherea hub provides 1-to-N switching function. The communication assembly C(603) includes a point-to-point (i.e., N-to-N) connection based routingelement C (613). As noted above, the routing element A (611), routingelement B (612), and routing element C (613) may be based on coppertrace based electrical connection and/or light guide based opticalconnection.

FIG. 7 depicts a flowchart of a method in accordance with one or moreembodiments of the invention. In one or more embodiments of theinvention, one or more of the steps shown in FIG. 2 may be omitted,repeated, and/or performed in a different order. Accordingly,embodiments of the invention should not be considered limited to thespecific arrangements of steps shown in FIG. 2. In one or moreembodiments, the method described in reference to FIG. 2 may bepracticed using the system (100).

Initially, in Step 201, pluggable modules external to a device areconnected to connectors of the device. Accordingly, the pluggablemodules are confined within a pre-determined 3D range defined at leastby the layout of the connectors on a mechanical enclosure of the device.In particular, the pre-determined 3D range is substantially external tothe device. In one or more embodiments of the invention, the pluggablemodules are held by the connectors in a rigid relative position to thedevice with mechanical stability.

In Step 202, signals are exchanged among the pluggable modules and thedevice via the connectors and a backplane internal to the device. Inparticular, the backplane and internal circuits of the device form anin-band channel of the device. In one or more embodiments of theinvention, the device is a node in a computer network and the signalsare network supplemental signals based on a pre-determined communicationprotocol, such as a network communication protocol.

In Step 203, a supplemental signal is exchanged among the pluggablemodules via a communication medium within the pre-determined 3D range.Accordingly, the pluggable modules communicate with each other bypassingthe connectors and the backplane. In one or more embodiments of theinvention, the communication medium includes a communication assemblyperipheral to the device and in proximity of the connectors. In one ormore embodiments of the invention, the communication medium includes thefree space for optical or other wireless communication within thepre-determined 3D range. In one or more embodiments of the invention,the supplemental signal provides a common frequency and phase tosynchronize the pluggable modules. In particular, the synchronizationerror is substantially less than the communication latency among thepluggable modules, in part due to the limited travel of the supplementalsignal within the pre-determined 3D range and/or due to thesubstantially reduced buffering and elimination of multiple trafficstreams competing for buffering resources of the supplemental signalpath.

In Step 204, the communication latency among the pluggable modules isreduced or otherwise compensated by using the supplemental signal toenhance the pre-determined communication protocol. In particular, thecommunication latency for a packet stream traveling through the devicevia any ingress and egress pluggable modules is compensated. In one ormore embodiments of the invention, reducing or otherwise compensatingthe communication latency includes the following steps.

(a) A global timing packet stream of the computer network is received bya pluggable module (referred to as the ingress pluggable module) from anetwork path leading to the device in the computer network.

(b) The ingress pluggable module performs ingress timestamping using thecommon frequency and phase of the supplemental signal. For example, aningress timestamp is inserted into the global timing packet stream. Theingress timestamp represents or otherwise corresponds to a time pointwhen the timing packet stream is received by the ingress pluggablemodule. The ingress timestamping is performed before the ingresspluggable module passes the global timing packet stream to anotherpluggable module (referred to as the egress pluggable module) throughthe in-band channel of the device.

(c) The egress pluggable module receives the global timing packet streamand performs egress timestamping using the common frequency and phase ofthe supplemental signal. For example, an egress time stamp is insertedinto the global timing packet stream. The egress timestamp represents orotherwise corresponds to a time point when the timing packet stream isreceived by the egress pluggable module through the in-band channel ofthe device.

(d) The ingress timestamp and egress timestamp are extracted by theegress pluggable module from the global timing packet. The extractedingress timestamp and egress timestamp are then subtracted or otherwisecompared to generate a difference, referred to as the residence time.

(e) The egress pluggable module uses the residence time to compensatethe communication latency. In one or more embodiments, the egresspluggable module compensates the communication latency by updating acorrection field in the global timing packet stream. Specifically, theresidence time stored in the correction field is used to reduce, offset,or otherwise mitigate the communication latency or an effect of thecommunication latency, e.g., by a downstream device receiving, directlyor indirectly, the global timing packet stream from the egress pluggablemodule. In one or more embodiments, the egress pluggable modulecompensates the communication latency by using the residence time toreduce, offset, or otherwise mitigate the communication latency or aneffect of the communication latency in the egress data packet streamreceived through the in-band channel of the device. For example, thedown stream device or the egress pluggable module may re-synchronize theegress data packet stream based on the residence time.

Embodiments of the invention may be implemented on virtually any type ofcomputing system, regardless of the platform being used. For example,the computing system may be one or more mobile devices (e.g., laptopcomputer, smart phone, personal digital assistant, tablet computer, orother mobile device), desktop computers, servers, blades in a serverchassis, or any other type of computing device or devices that includesat least the minimum processing power, memory, and input and outputdevice(s) to perform one or more embodiments of the invention.

For example, as shown in FIG. 8, the computing system (800) may includeone or more computer processor(s) (802), associated memory (804) (e.g.,random access memory (RAM), cache memory, flash memory, etc.), one ormore storage device(s) (806) (e.g., a hard disk, an optical drive suchas a compact disk (CD) drive or digital versatile disk (DVD) drive, aflash memory stick, etc.), and numerous other elements andfunctionalities. The computer processor(s) (802) may be an integratedcircuit for processing instructions. For example, the computerprocessor(s) may be one or more cores, or micro-cores of a processor.The computing system (800) may also include one or more input device(s)(810), such as a touchscreen, keyboard, mouse, microphone, touchpad,electronic pen, or any other type of input device. Further, thecomputing system (800) may include one or more output device(s) (808),such as a screen (e.g., a liquid crystal display (LCD), a plasmadisplay, touchscreen, cathode ray tube (CRT) monitor, projector, orother display device), a printer, external storage, or any other outputdevice. One or more of the output device(s) may be the same or differentfrom the input device(s). The input and output device(s) may be locallyor remotely connected to the computer processor(s) (802), memory (804),and storage device(s) (806). Many different types of computing systemsexist, and the aforementioned input and output device(s) may take otherforms.

Software instructions in the form of computer readable program code toperform embodiments of the invention may be stored, in whole or in part,temporarily or permanently, on a non-transitory computer readable mediumsuch as a CD, DVD, storage device, a diskette, a tape, flash memory,physical memory, or any other non-transitory computer readable storagemedium. Specifically, the software instructions may correspond tocomputer readable program code that when executed by a processor(s), isconfigured to perform embodiments of the invention.

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of thisdisclosure, will appreciate that other embodiments can be devised whichdo not depart from the scope of the invention as disclosed herein.Accordingly, the scope of the invention should be limited only by theattached claims.

What is claimed is:
 1. A pluggable module comprising: an interfaceconfigured to connect to a connector associated with a device havinginternal circuits; an optical interface configured to connect to anetwork path, to enable communication between the internal circuits andthe network path; and a signal interface configured to connect to asecond pluggable module in the device, wherein the signal interfaceconnects to the second pluggable module as an out-of-band channel thatoperates independently from a backplane of the device and the internalcircuits.
 2. The pluggable module of claim 1, wherein the out-of-bandchannel bypasses the device.
 3. The pluggable module of claim 1, whereinthe pluggable module communicates with the second pluggable module viaan in-band channel through the device and via the out-of-band channel.4. The pluggable module of claim 3, wherein the out-of-band channel is asupplemental signal used for timing synchronization between thepluggable module and the second pluggable module.
 5. The pluggablemodule of claim 1, wherein the signal interface is a wireless interface.6. The pluggable module of claim 5, wherein the wireless interface isone of Bluetooth, Wi-Fi, and Near Field Communication (NFC).
 7. Thepluggable module of claim 1, wherein the signal interface is aproximity-based connection with the second pluggable module.
 8. Thepluggable module of claim 7, wherein the proximity-based connection withthe second pluggable module is via capacitive coupling using a groundconnection to the device.
 9. The pluggable module of claim 1, whereinthe signal interface is a wired connection.
 10. A device comprising: abackplane; internal circuits communicatively connected to the backplane;one or more connectors communicatively coupled to the internal circuitsand the backplane, wherein the one or more connectors are eachconfigured to connect to a pluggable module of a plurality of pluggablemodules, wherein the plurality of pluggable modules are configured tocommunicate to one another via an out-of-band channel that operatesindependently from the device.
 11. The device of claim 10, wherein theout-of-band channel bypasses the device and an in-band channel isutilized through the device for communication between any of theplurality of pluggable modules.
 12. The device of claim 10, wherein theout-of-band channel is a supplemental signal used for timingsynchronization between a first pluggable module and a second pluggablemodule.
 13. The device of claim 10, wherein the out-of-band channel isvia a wireless connection.
 14. The device of claim 13, wherein thewireless connection is one of Bluetooth, Wi-Fi, and Near FieldCommunication (NFC).
 15. The device of claim 10, wherein the out-of-bandchannel is via a proximity-based connection.
 16. The device of claim 15,wherein the proximity-based connection is via capacitive coupling usinga ground connection between the plurality of pluggable modules and thedevice.
 17. The device of claim 10, wherein the out-of-band channel isvia a wired connection.
 18. A method comprising: operating a device witha plurality of pluggable modules; communicating between two or morepluggable modules of the plurality of pluggable modules via an in-bandchannel through the device; and communicating between the two or morepluggable modules via an out-of-band channel that operates independentlyfrom and bypasses the device.
 19. The method of claim 18, wherein theout-of-band channel is via a wireless connection.
 20. The method ofclaim 18, wherein the out-of-band channel is via a proximity-basedconnection.